Vehicle control device, vehicle control method, and storage medium

ABSTRACT

A vehicle control device that can appropriately recognize a direction of a vehicle is provided. The vehicle control device includes: an image acquirer configured to acquire an image obtained by imaging a space outside of a vehicle; an object detector configured to detect, through image processing, a plurality of types of objects including a road structure and a moving object shown in the image; a reference line setter configured to select one type of object from the plurality of types of objects and to set a reference line based on a direction of the selected type of object; and a vehicle direction estimator configured to estimate an angle which is formed by the reference line and a traveling direction line of the vehicle as a direction of the vehicle relative to a lane in which the vehicle is traveling or is scheduled to travel.

CROSS-REFERENCE TO RELATED APPLICATION

The application is based on Japanese Patent Application No. 2021-024365filed on Feb. 18, 2021, the content of which incorporated herein byreference.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a vehicle control device, a vehiclecontrol method, and a storage medium.

Description of Related Art

A technique of detecting information of another vehicle relative to ahost vehicle is known. For example, Japanese Unexamined PatentApplication, First Publication No. 2017-161430 discloses a technique ofdetecting a relative position of another vehicle based on a radio signalwhich is transmitted from the other vehicle.

SUMMARY OF THE INVENTION

However, when only relative information such as a relative position ofanother vehicle is used, for example, a direction of a host vehicle maynot be appropriately recognized at the time of lane change of the hostvehicle.

The invention has been made in view of the above-mentioned circumstancesand an objective thereof is to provide a vehicle control device, avehicle control method, and a storage medium storing a program that canappropriately recognize a direction of a vehicle.

A vehicle control device according to the invention employs thefollowing configurations.

(1) According to an aspect of the invention, there is provided a vehiclecontrol device including: an image acquirer configured to acquire animage obtained by imaging a space outside of a vehicle; an objectdetector configured to detect, through image processing, a plurality oftypes of objects including a road structure and a moving object shown inthe image; a reference line setter configured to select one type ofobject from the plurality of types of objects and to set a referenceline based on a direction of the selected type of object; and a vehicledirection estimator configured to estimate an angle which is formed bythe reference line and a traveling direction line of the vehicle as adirection of the vehicle relative to a lane in which the vehicle istraveling or is scheduled to travel.

(2) In the aspect of (1), when the selected type of object is the roadstructure, the direction of the selected type of object is an extendingdirection of the road structure, or when the selected type of object isthe moving object, the direction of the selected type of object is atraveling direction of the moving object.

(3) In the aspect of (1), the reference line setter may select one typeof object from among the detected plurality of types of objects based ona predetermined priority.

(4) In the aspect of (3), in the predetermined priority, a boundary lineof the lane in which the vehicle is traveling or is scheduled to travelmay have the highest priority, a boundary line of an adjacent lane ofthe lane may have the second highest priority, and another vehicle nearthe vehicle may have the third highest priority.

(5) In the aspect of (1), the vehicle control device may further includea driving controller configured to generate a target trajectory based onthe direction of the vehicle estimated by the vehicle directionestimator and to control steering and acceleration/deceleration of thevehicle regardless of operation of a driver of the vehicle such that thevehicle travels along the generated target trajectory.

(6) According to another aspect of the invention, there is provided avehicle control method which is performed by a computer mounted on avehicle, the vehicle control method including: acquiring an imageobtained by imaging a space outside of a vehicle; detecting, throughimage processing, a plurality of types of objects including a roadstructure and a moving object shown in the image; selecting one type ofobject from the plurality of types of objects and setting a referenceline based on a direction of the selected type of object; and estimatingan angle which is formed by the reference line and a traveling directionline of the vehicle as a direction of the vehicle relative to a lane inwhich the vehicle is traveling or is scheduled to travel.

(7) According to another aspect of the invention, there is provided acomputer-readable non-transitory storage medium storing a program forcausing a computer mounted on a vehicle to: acquire an image obtained byimaging a space outside of a vehicle; detect, through image processing,a plurality of types of objects including a road structure and a movingobject shown in the image; select one type of object from the pluralityof types of objects and set a reference line based on a direction of theselected type of object; and estimate an angle which is formed by thereference line and a traveling direction line of the vehicle as adirection of the vehicle relative to a lane in which the vehicle istraveling or is scheduled to travel.

According to the aspects of (1) to (7), it is possible to appropriatelyrecognize a direction of a vehicle.

According to the aspect of (5), it is possible to appropriately controlthe vehicle based on the recognized direction of the vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a configuration of a vehicle systemincluding a vehicle control device according to an embodiment;

FIG. 2 is a diagram illustrating functional configurations of a firstcontroller and a second controller;

FIGS. 3A and 3B are diagrams illustrating an example of travel controlof a vehicle according to a comparative example;

FIG. 4 is a diagram illustrating an example of a priority with which areference line setter selects an object;

FIG. 5 is a diagram illustrating an example of a first situation inwhich setting of a reference line RL and estimation of a direction of ahost vehicle M are performed;

FIG. 6 is a diagram illustrating an example of a second situation inwhich setting of a reference line RL and estimation of a direction of ahost vehicle M are performed;

FIG. 7 is a diagram illustrating an example of a third situation inwhich setting of a reference line RL and estimation of a direction of ahost vehicle M are performed;

FIG. 8 is a diagram illustrating an example of a fourth situation inwhich setting of a reference line RL and estimation of a direction of ahost vehicle M are performed; and

FIG. 9 is a flowchart illustrating an example of a flow of processeswhich are performed in cooperation by a camera, an object recognitiondevice, and an automated driving control device.

DETAILED DESCRIPTION OF THE INVENTION

Hereinafter, a vehicle control device, a vehicle control method, and astorage medium according to an embodiment of the invention will bedescribed with reference to the accompanying drawings.

Overall Configuration

FIG. 1 is a diagram illustrating a configuration of a vehicle system 1including a vehicle control device according to an embodiment. A vehiclein which the vehicle system 1 is mounted is, for example, a vehicle withtwo wheels, three wheels, or four wheels and a drive source thereof isan internal combustion engine such as a diesel engine or a gasolineengine, an electric motor, or a combination thereof. An electric motoroperates using electric power which is generated by a power generatorconnected to the internal combustion engine or electric power which isdischarged from a secondary battery or a fuel cell.

The vehicle system 1 includes, for example, a camera 10, a radar device12, a Light Detection and Ranging device (LIDAR) 14, an objectrecognition device 16, a communication device 20, a human-machineinterface (HMI) 30, a vehicle sensor 40, a navigation device 50, a mappositioner (MPU) 60, a driving operator 80, an automated driving controldevice 100, a travel driving force output device 200, a brake device210, and a steering device 220. These devices or instruments areconnected to each other via a multiplex communication line such as acontroller area network (CAN) communication line, a serial communicationline, a radio communication network, or the like. The configurationillustrated in FIG. 1 is only an example and a part of the configurationmay be omitted or another configuration may be added thereto.

The camera 10 is, for example, a digital camera using a solid-stateimaging device such as a charge coupled device (CCD) or a complementarymetal oxide semiconductor (CMOS). The camera 10 is attached to anarbitrary position on a vehicle (hereinafter, referred to as a hostvehicle M) in which the vehicle system 1 is mounted. For example, whenthe front view of the vehicle M is imaged, the camera 10 is attached toan upper part of a front windshield, a rear surface of a rearviewmirror, or the like. The camera 10 images surroundings of the hostvehicle M, for example, periodically and repeatedly. The camera 10 maybe a stereoscopic camera.

The radar device 12 radiates radio waves such as millimeter waves to thesurroundings of the host vehicle M, detects radio waves (reflectedwaves) reflected by an object, and detects at least a position (adistance and a direction) of the object. The radar device 12 is attachedto an arbitrary position on the host vehicle M. The radar device 12 maydetect a position and a speed of an object using a frequency modulatedcontinuous wave (FM-CW) method.

The LIDAR 14 applies light to the surroundings of the host vehicle M andmeasures scattered light. The LIDAR 14 detects a distance to an objectbased on a time from emission of light to reception of light. The lightwhich is applied is, for example, a pulse-like laser beam. The LIDAR 14is attached to an arbitrary position on the host vehicle M.

The object recognition device 16 performs a sensor fusion process onresults of detection from some or all of the camera 10, the radar device12, and the LIDAR 14 and recognizes a position, a type, a speed, and thelike of an object. The object recognition device 16 outputs the resultof recognition to the automated driving control device 100. The objectrecognition device 16 may output the results of detection from thecamera 10, the radar device 12, and the LIDAR 14 to the automateddriving control device 100 without any change. The object recognitiondevice 16 may be omitted from the vehicle system 1. In this embodiment,the object recognition device 16 includes an image acquirer 16A and anobject detector 16B. The image acquirer 16A acquires an image of a spaceoutside of the vehicle which is captured by the camera 10. The objectdetector 16B detects, through image processing, a plurality of types ofobjects including a road structure and a moving object shown in theimage.

The communication device 20 communicates with other vehicles near thehost vehicle M, for example, using a cellular network, a Wi-Fi network,Bluetooth (registered trademark), or dedicated short range communication(DSRC) or communicates with various server devices via a radio basestation.

The HMI 30 presents various types of information to an occupant of thehost vehicle M and receives an input operation from the occupant. TheHMI 30 includes various display devices, speakers, buzzers, a touchpanel, switches, and keys.

The vehicle sensor 40 includes a vehicle speed sensor that detects aspeed of the host vehicle M, an acceleration sensor that detectsacceleration, a yaw rate sensor that detects an angular velocity arounda vertical axis, and a direction sensor that detects a direction of thehost vehicle M.

The navigation device 50 includes, for example, a global navigationsatellite system (GNSS) receiver 51, a navigation HMI 52, and a routedeterminer 53. The navigation device 50 stores first map information 54in a storage device such as a hard disk drive (HDD) or a flash memory.The GNSS receiver 51 identifies a position of the host vehicle M basedon signals received from GNSS satellites. The position of the hostvehicle M may be identified or complemented by an inertial navigationsystem (INS) using the output of the vehicle sensor 40. The navigationHMI 52 includes a display device, a speaker, a touch panel, and keys. Awhole or a part of the navigation HMI 52 may be shared by the HMI 30.For example, the route determiner 53 determines a route (hereinafter,referred to as a route on a map) from the position of the host vehicle Midentified by the GNSS receiver 51 (or an input arbitrary position) to adestination input by an occupant using the navigation HMI 52 withreference to the first map information 54. The first map information 54is, for example, information in which road shapes are expressed by linksindicating roads and nodes connected by the links. The first mapinformation 54 may include a curvature of a road or point of interest(POI) information. The route on a map is output to the MPU 60. Thenavigation device 50 may perform guidance for a route using thenavigation HMI 52 based on the route on a map. The navigation device 50may be realized, for example, by a function of a terminal device such asa smartphone or a tablet terminal which is carried by an occupant. Thenavigation device 50 may transmit a current position and a destinationto a navigation server via the communication device 20 and acquire aroute which is equivalent to the route on a map from the navigationserver.

The MPU 60 includes, for example, a recommended lane determiner 61 andstores second map information 62 in a storage device such as an HDD or aflash memory. The recommended lane determiner 61 divides a route on amap supplied from the navigation device 50 into a plurality of blocks(for example, every 100 [m] in a vehicle travel direction) anddetermines a recommended lane for each block with reference to thesecond map information 62. The recommended lane determiner 61 determinesin which lane from the leftmost the host vehicle is to travel. Whenthere is a branching point in the route on a map, the recommended lanedeterminer 61 determines a recommended lane such that the host vehicle Mtravels on a rational route for traveling to a branching destination.

The second map information 62 is map information with higher precisionthan that of the first map information 54. The second map information 62includes, for example, information on the centers of lanes orinformation on boundaries of lanes. The second map information 62 mayinclude road information, traffic regulation information, addressinformation (addresses and postal codes), facility information, andphone number information. The second map information 62 may be updatedfrom time to time by causing the communication device 20 to communicatewith another device.

The driving operator 80 includes, for example, an accelerator pedal, abrake pedal, a shift lever, a steering wheel, a deformed steering wheel,a joystick, and other operators. A sensor that detects an amount ofoperation or performing of an operation is attached to the drivingoperator 80, and results of detection thereof are output to theautomated driving control device 100 or some or all of the traveldriving force output device 200, the brake device 210, and the steeringdevice 220.

The automated driving control device 100 includes, for example, a firstcontroller 120 and a second controller 160. The first controller 120 andthe second controller 160 are realized, for example, by causing ahardware processor such as a central processor (CPU) to execute aprogram (software). Some or all of such elements may be realized inhardware (which includes circuitry) such as a large scale integration(LSI), an application specific integrated circuit (ASIC), or afield-programmable gate array (FPGA), or a graphics processor (GPU) ormay be realized in cooperation of software and hardware. The program maybe stored in a storage device (a storage device including anon-transitory storage medium) such as an HDD or a flash memory of theautomated driving control device 100 in advance, or may be installed inthe HDD or the flash memory of the automated driving control device 100by storing the program in a removable storage medium (a non-transitorystorage medium) such as a DVD or a CD-ROM and attaching the removablestorage medium to a drive device. A combination of the objectrecognition device 16 and the automated driving control device 100 arean example of a “vehicle control device,” and a combination of amovement plan creator 140 and the second controller 160 are an exampleof a “driving controller.”

FIG. 2 is a diagram illustrating functional configurations of the firstcontroller 120 and the second controller 160. The first controller 120includes, for example, a recognizer 130 and a movement plan creator 140.The first controller 120 is realized, for example, using a functionbased on artificial intelligence (AI) and a function based on apredetermined model together. For example, a function of “recognizing acrossing” may be embodied by performing recognition of a crossing basedon deep learning or the like and recognition based on predeterminedconditions (such as signals and road signs which can bepattern-matched), scoring both recognitions, and comprehensivelyevaluating both recognitions. Accordingly, reliability of automateddriving is secured.

The recognizer 130 recognizes states such as a position, a speed, and anacceleration of an object near the host vehicle M based on informationinput from the camera 10, the radar device 12, and the LIDAR 14 via theobject recognition device 16. For example, a position of an object isrecognized as a position in an absolute coordinate system with an originset to a representative point of the host vehicle M (such as the centerof gravity or the center of a drive shaft) and is used for control. Aposition of an object may be expressed as a representative point such asthe center of gravity or a corner of the object or may be expressed as adrawn area. A “state” of an object may include an acceleration or a jerkof the object or a “moving state” (for example, whether lane change isbeing performed or whether lane change is going to be performed)thereof.

The recognizer 130 recognizes, for example, a lane (a travel lane) inwhich the host vehicle M is traveling. For example, the recognizer 130recognizes the travel lane by comparing a pattern of road marking lines(hereinafter referred to as boundary lines) near the host vehicle Mwhich are recognized from an image captured by the camera 10 with apattern of boundary lines (for example, arrangement of a solid line anda dotted line) which are acquired from the second map information 62.The recognizer 130 is not limited to the boundary lines, but mayrecognize the travel lane by recognizing travel road boundaries (roadboundaries) including boundary lines, edges of roadsides, curbstones,medians, and guard rails. In this recognition, the position of the hostvehicle M acquired from the navigation device 50 and the result ofprocessing from the INS may be considered. The recognizer 130 recognizesa stop line, an obstacle, a red signal, a toll gate, or other roadevents.

The recognizer 130 further includes a reference line setter 130A and avehicle direction estimator 130B, and estimates the direction of thehost vehicle M with respect to the reference line set by the referenceline setter 130A. Details of the functions of the reference line setter130A and the vehicle direction estimator 130B will be described later.

The movement plan creator 140 creates a target trajectory in which thehost vehicle M will travel autonomously (without requiring a driver'soperation) in the future such that the host vehicle M travels in arecommended lane determined by the recommended lane determiner 61 inprinciple and copes with surrounding circumstances of the host vehicle Mbased on the direction of the host vehicle estimated by the vehicledirection estimator 130B. A target trajectory includes, for example, aspeed element. For example, a target trajectory is expressed bysequentially arranging points (trajectory points) at which the hostvehicle M is to arrive. Trajectory points are points at which the hostvehicle M is to arrive at intervals of a predetermined travelingdistance (for example, about several [m]) along a road, and a targetspeed and a target acceleration at intervals of a predetermined samplingtime (for example, about below the decimal point [sec]) are created as apart of a target trajectory in addition. Trajectory points may bepositions at which the host vehicle M is to arrive at sampling timesevery predetermined sampling time. In this case, information of a targetspeed or target acceleration is expressed by intervals between thetrajectory points.

The movement plan creator 140 may set events of automated driving increating a target trajectory. The events of automated driving include aconstant-speed travel event, a low-speed following travel event, a lanechange event, a branching event, a merging event, and a take-over event.The movement plan creator 140 creates a target trajectory based onevents which are started.

The second controller 160 controls the travel driving force outputdevice 200, the brake device 210, and the steering device 220 such thatthe host vehicle M travels along a target trajectory created by themovement plan creator 140 as scheduled.

Referring to FIG. 2, the second controller 160 includes, for example, anacquirer 162, a speed controller 164, and a steering controller 166. Theacquirer 162 acquires information of a target trajectory (trajectorypoints) created by the movement plan creator 140 and stores the acquiredinformation in a memory (not illustrated). The speed controller 164controls the travel driving force output device 200 or the brake device210 based on a speed element accessory to the target trajectory storedin the memory. The steering controller 166 controls the steering device220 based on a curve state of the target trajectory stored in thememory. The processes of the speed controller 164 and the steeringcontroller 166 are realized, for example, in combination of feed-forwardcontrol and feedback control. For example, the steering controller 166performs control in combination of feed-forward control based on acurvature of a road in front of the host vehicle M and feedback controlbased on separation from the target trajectory.

The travel driving force output device 200 outputs a travel drivingforce (a torque) for allowing the vehicle to travel to the drivingwheels. The travel driving force output device 200 includes, forexample, a combination of an internal combustion engine, an electricmotor, and a transmission and an ECU that controls them. The ECUcontrols the elements based on information input from the secondcontroller 160 or information input from the driving operator 80.

The brake device 210 includes, for example, a brake caliper, a cylinderthat transmits a hydraulic pressure to the brake caliper, an electricmotor that generates a hydraulic pressure in the cylinder, and a brakeECU. The brake ECU controls the electric motor based on the informationinput from the second controller 160 or the information input from thedriving operator 80 such that a brake torque based on a brakingoperation is output to vehicle wheels. The brake device 210 may includea mechanism for transmitting a hydraulic pressure generated by anoperation of a brake pedal included in the driving operator 80 to thecylinder via a master cylinder as a backup. The brake device 210 is notlimited to the above-mentioned configuration, and may be anelectronically controlled hydraulic brake device that controls anactuator based on information input from the second controller 160 suchthat the hydraulic pressure of the master cylinder is transmitted to thecylinder.

The steering device 220 includes, for example, a steering ECU and anelectric motor. The electric motor changes a direction of turningwheels, for example, by applying a force to a rack-and-pinion mechanism.The steering ECU drives the electric motor based on the informationinput from the second controller 160 or the information input from thedriving operator 80 to change the direction of the turning wheels.

COMPARATIVE EXAMPLES

A comparative example will be described below with reference to FIGS. 3Aand 3B. FIGS. 3A and 3B are diagrams illustrating an example oftraveling control of a vehicle m according to a comparative example. Thevehicle m has an automated driving function and does not include atleast the reference line setter 130A and the vehicle direction estimator130B illustrated in FIG. 2. In FIGS. 3A and 3B, the vehicle m istraveling in a lane L1 and is scheduled to perform lane change to a laneL2. Since vehicles m1 and m2 which are neighboring vehicles of thevehicle m are traveling straight ahead in lanes L2 and L3, respectively,the vehicle m can perform lane change without any problems bymaintaining or increasing a speed at the time of lane change to the laneL2.

On the other hand, in FIG. 3B, the vehicle m is traveling in the lane L1and is scheduled to perform lane change to the lane L2. At this time,relative positional relationship between the vehicle m and the vehiclesm1 and m2 are similar to those in FIG. 3A. However, in FIG. 3B, thevehicles m1 and m2 are traveling in directions in which they will departfrom the lanes L2 and L3, respectively, and thus, particularly, lanechange of the vehicle m to the lane L2 may cause a problem with thevehicle m2. That is, it may not be appropriate to control the vehicle mbased on only the relative positional relationships between the vehiclem and the vehicles m1 and m2.

Between FIG. 3A and FIG. 3B, the relative positional relationshipsbetween the vehicle m and the vehicles m1 and m2 are similar to eachother, but an angle which is formed by the traveling direction of thevehicle m and a boundary line of the lane L1 in which the vehicle m istraveling is different. Specifically, in FIG. 3A, an angle which isformed by the traveling direction of the vehicle m and an extendingdirection of the boundary line of the lane L1, that is, the direction ofthe vehicle m, has a positive value. In FIG. 3B, the direction of thevehicle m is substantially zero. Accordingly, in the situationillustrated in FIG. 3B, since the direction of the vehicle m issubstantially zero, another vehicle instead of the vehicle m istraveling in a direction in which it will depart from a lane and maycause a problem in lane change of the vehicle m. On the other hand, thehost vehicle M according to this embodiment can set a reference line andestimate the direction of the host vehicle M with respect to thereference line, whereby it is possible to more appropriately controltraveling of the host vehicle M. Details thereof will be describedbelow.

Setting of reference line and estimation of direction of host vehicle M

The reference line setter 130A selects one type of object based on apredetermined priority from a plurality of types of objects recognizedby the object recognition device 16 and sets a virtual reference line RL(which corresponds to the “extending direction of the boundary line”)corresponding to a direction of the selected type of object. At thistime, the objects recognized by the object recognition device 16 areexpressed in a camera coordinate system with an origin set to the camera10, and the reference line setter 130A converts the coordinates in thecamera coordinate system to coordinates on an assumed plane which isexpressed as a two-dimensional plane when the surrounding space of thehost vehicle M is seen from the sky and then sets the reference line RL.

The vehicle direction estimator 130B estimates an angle which is formedby the set reference line RL and a traveling direction line of the hostvehicle M (which corresponds to the “traveling direction of the vehiclem”) as the direction of the host vehicle M with respect to a lane inwhich the host vehicle M is traveling or is scheduled to travel. In thisembodiment, the traveling direction line is a central axis of the hostvehicle M, or may be substantially, for example, an instantaneous movingdirection of the host vehicle M, that is, the actual traveling directionof the host vehicle M.

FIG. 4 is a diagram illustrating an example of a priority with which thereference line setter 130A selects an object. As illustrated in FIG. 4,a boundary line, a guard rail, and a wall of a host lane are set toPriority 1 indicating the highest priority. The boundary line, the guardrail, and the wall of the host lane may be those of a lane in which thehost vehicle M is traveling or those in a lane in which the host vehicleM is scheduled to travel. For example, immediately before the hostvehicle M performs lane change or while the host vehicle M is performinglane change, a boundary line, a guard rail, or a wall of a lane which isa change destination may be selected as an object instead of theboundary line, the guard rail, or the wall of the lane in which the hostvehicle is traveling. When two or more types of objects out of theboundary line, the guard rail, and the wall of the host lane arerecognized by the object recognition device 16, the reference linesetter 130A can select one type of object using an arbitrary method and,for example, may select an object closest to the host vehicle M. Some orall of the boundary line, the guard rail, and the wall are an example ofa “road structure.”

A boundary line, a guard rail, and a wall of an adjacent lane are set toPriority 2 indicating the second highest priority. That is, when theboundary line, the guard rail, and the wall of the host lane are notincluded in the objects recognized by the object recognition device 16,the reference line setter 130A selects the boundary line, the guardrail, or the wall of the adjacent lane as an object. When two or moretypes of objects out of the boundary line, the guard rail, and the wallof the adjacent lane are recognized by the object recognition device 16,the reference line setter 130A can select one type of object using anarbitrary method and, for example, may select an object closest to thehost vehicle M.

Another vehicle in an adjacent lane is set to Priority 3 indicating thethird highest priority. That is, when the boundary lines, the guardrails, and the walls of the host lane and the adjacent lane are notincluded in the objects recognized by the object recognition device 16,the reference line setter 130A selects another vehicle in the adjacentlane as an object. Here, when two or more vehicles in the adjacent laneare recognized, the reference line setter 130A selects the two or morevehicles as objects. Another vehicle in an adjacent lane is an exampleof a “moving object.”

In this way, the reference line setter 130A selects one type of objectbased on the priority illustrated in FIG. 4, and sets the reference lineRL corresponding to the direction of the selected type of object. Here,a “direction of an object” indicates an extending direction when theobject is a boundary line, a guard rail, or a wall, and indicates amoving direction of another vehicle when the object is another vehiclein an adjacent lane. In this case, the moving direction may be a centralaxis of another vehicle or may be an estimated moving direction ofanother vehicle. When two or more objects of one type are selected orwhen two or more vehicles in an adjacent lane are selected, thereference line setter 130A sets the reference line RL corresponding toan average direction of the two or more objects.

Situation Examples

Examples of situations in which setting of a reference line RL andestimation of a direction of a host vehicle M are performed will bedescribed below with reference to FIGS. 5 to 8. FIG. 5 is a diagramillustrating an example of a first situation in which setting of thereference line RL and estimation of the direction of the host vehicle Mare performed. In FIG. 5, the host vehicle M is traveling in a lane L1and is going to perform lane change to a lane L2 due to a decrease inthe number of lanes. In this situation, the camera 10 acquires an imageof a boundary line BL1 and a boundary line BL2 of the lane L1, aboundary line BL3 of the lane L2, and another vehicle M1, and the objectrecognition device 16 detects the boundary line BL1, the boundary lineBL2, the boundary line BL3, and the other vehicle M1 as objects throughimage processing. Then, the reference line setter 130A selects theboundary line BL1 and the boundary line BL2 with Priority 1 as objectsbased on the priority illustrated in FIG. 4, and sets a reference lineRL corresponding to an average direction of the boundary line BL1 andthe boundary line BL2. Then, the vehicle direction estimator 130Bestimates an angle θ which is formed by the set reference line RL andthe traveling direction line of the host vehicle M as the direction ofthe host vehicle M with respect to the lane L1. Then, the movement plancreator 140 creates a target trajectory of the host vehicle M based onthe direction of the host vehicle estimated by the vehicle directionestimator 130B, and the second controller 160 controls steering andacceleration/deceleration of the host vehicle M regardless of operationby a driver of the host vehicle M such that the host vehicle M travelsalong the created target trajectory.

In the example of the first situation, the reference line setter 130Aselects the boundary line BL1 and the boundary line BL2 with Priority 1as objects and sets the reference line RL corresponding to the averagedirection thereof, and the reference line setter 130A may select onethereof as an object and set the reference line RL corresponding to thedirection thereof. For example, the boundary line closer to the hostvehicle M out of the boundary line BL1 and the boundary line BL2 may beselected as an object. The reference line setter 130A may recognize thelane L2 which is a lane after lane change instead of the boundary lineof the lane L1 which is a lane before lane change as a lane in which thehost vehicle M is scheduled to travel, and select the boundary line BL2thereof as an object.

FIG. 6 is a diagram illustrating an example of a second situation inwhich setting of the reference line RL and estimation of the directionof the host vehicle M are performed. In FIG. 6, the host vehicle M isperforming lane change from the lane L1 to the lane L2 and is enteringthe lane L2. In this situation, the camera 10 acquires an image of theboundary line BL1 and the boundary line BL3 of the lane L2, the boundaryline BL2 of the lane L1, and another vehicle M1, and the objectrecognition device 16 detects the boundary line BL1, the boundary lineBL3, the boundary line BL2, and the other vehicle M1 as objects throughimage processing. Then, the reference line setter 130A selects theboundary line BL1 and the boundary line BL3 with Priority 1 as objectsbased on the priority illustrated in FIG. 4, and sets the reference lineRL corresponding to an average direction of the boundary line BL1 andthe boundary line BL3. Then, the vehicle direction estimator 130Bestimates an angle θ which is formed by the set reference line RL andthe traveling direction line of the host vehicle M as the direction ofthe host vehicle M with respect to the lane L2. Then, the movement plancreator 140 creates a target trajectory of the host vehicle M based onthe direction of the host vehicle estimated by the vehicle directionestimator 130B, and the second controller 160 controls steering andacceleration/deceleration of the host vehicle M regardless of operationby a driver of the host vehicle M such that the host vehicle M travelsalong the created target trajectory. Similarly to the situationillustrated in FIG. 5, the reference line setter 130A may select one ofthe boundary line BL1 and the boundary line BL3 as an object and set thereference line RL corresponding to the direction thereof.

FIG. 7 is a diagram illustrating an example of a third situation inwhich setting of the reference line RL and estimation of the directionof the host vehicle M are performed. In FIG. 7, the host vehicle M isperforming lane change from the lane L1 to the lane L2. In thissituation, the camera 10 acquires an image of the boundary line BL3 ofthe lane L2 and another vehicle M1. In this case, since the boundaryline BL3 is thin and is intermittently formed, the object recognitiondevice 16 may not detect the boundary line BL3 as an object throughimage processing. Even when the object recognition device 16 detects theboundary line BL3 as an object, the intermittent boundary line may havelow accuracy as the reference line. Therefore, the reference line setter130A selects the other vehicle with Priority 3 as an object based on thepriority illustrated in FIG. 4, and sets the reference line RLcorresponding to the direction of the other vehicle M1. Then, thevehicle direction estimator 130B estimates an angle θ which is formed bythe set reference line RL and the traveling direction line of the hostvehicle M as the direction of the host vehicle M with respect to thelane L2. Then, the movement plan creator 140 creates a target trajectoryof the host vehicle M based on the direction of the host vehicleestimated by the vehicle direction estimator 130B, and the secondcontroller 160 controls steering and acceleration/deceleration of thehost vehicle M regardless of operation by a driver of the host vehicle Msuch that the host vehicle M travels along the created targettrajectory.

FIG. 8 is a diagram illustrating an example of a fourth situation inwhich setting of the reference line RL and estimation of the directionof the host vehicle M are performed. In FIG. 8, the host vehicle M istraveling in the lane L1 and is going to merge with the lane L2. In thissituation, the camera 10 acquires an image of the boundary line BL1 ofthe lane L1, the boundary line BL2 of the lane L2, a boundary line BL3of a lane L3, and another vehicle M1, and the object recognition device16 detects the boundary line BL1, the boundary line BL2, the boundaryline BL3, and the other vehicle M1 as objects through image processing.Then, the reference line setter 130A selects the boundary line BL1 withPriority 1 as an object based on the priority illustrated in FIG. 4, andsets a reference line RL corresponding to an average direction of theboundary line BL1. Then, the vehicle direction estimator 130B estimatesan angle θ which is formed by the set reference line RL and thetraveling direction line of the host vehicle M as the direction of thehost vehicle M with respect to the lane L1. Then, the movement plancreator 140 creates a target trajectory of the host vehicle M based onthe direction of the host vehicle estimated by the vehicle directionestimator 130B, and the second controller 160 controls steering andacceleration/deceleration of the host vehicle M regardless of operationby a driver of the host vehicle M such that the host vehicle M travelsalong the created target trajectory. Similarly to the exampleillustrated in FIG. 5, the reference line setter 130A may recognize thelane L2 which is a lane after merging instead of the boundary line ofthe lane L1 which is a lane before merging as a lane in which the hostvehicle M is scheduled to travel, and select the boundary line BL2thereof as an object.

Flow of Operations

A flow of processes which are performed in cooperation by the camera 10,the object recognition device 16, and the automated driving controldevice 100 will be described below with reference to FIG. 9. FIG. 9 is aflowchart illustrating an example of a flow of processes which areperformed in cooperation by the camera 10, the object recognition device16, and the automated driving control device 100. The flow of processesof the flowchart is repeatedly performed at intervals of a predeterminedcontrol cycle while the host vehicle M is traveling.

First, the camera 10 acquires an image obtained by imaging a spaceoutside of the host vehicle M (S100). Then, the object recognitiondevice 16 detects, through image processing, a plurality of types ofobjects including a road structure and a moving object shown in theimage acquired by the camera 10 (S101). Then, the reference line setter130A of the automated driving control device 100 selects one type ofobject from the detected plurality of types of objects based on thepriority illustrated in FIG. 4 (S102). Then, the reference line setter130A of the automated driving control device 100 sets a reference linecorresponding to a direction of the selected type of object (S103).Then, the vehicle direction estimator 130B of the automated drivingcontrol device 100 estimates an angle which is formed by the setreference line and the traveling direction line of the host vehicle M asthe direction of the host vehicle M with respect to a lane in which thehost vehicle M is traveling or is scheduled to travel (S104). Then, thedriving controller of the automated driving control device 100 generatesa target trajectory based on the direction of the vehicle estimated bythe vehicle direction estimator 130B and controls steering andacceleration/deceleration of the host vehicle M regardless of operationof a driver of the host vehicle M such that the host vehicle travelsalong the generated target trajectory (S105). Accordingly, the flow ofprocesses of the flowchart ends.

As described above, according to this embodiment, objects included in animage acquired by the camera 10 are detected, one type of object isselected from among the detected objects based on a predeterminedpriority, a reference line corresponding to the direction of theselected object is set, and an angle which is formed by the setreference line and the traveling direction line of the host vehicle M isestimated as the direction of the host vehicle M. Accordingly, it ispossible to appropriately recognize a direction of a vehicle.

The above-mentioned embodiment can be expressed as follows:

A vehicle control device including:

a storage device that stores a program; and

a hardware processor,

wherein the hardware processor is configured to execute the programstored in the storage device to:

acquire an image obtained by imaging a space outside of a vehicle;

detect, through image processing, a plurality of types of objectsincluding a road structure and a moving object shown in the image;

select one type of object from the plurality of types of objects and seta reference line based on a direction of the selected type of object;and

estimate an angle which is formed by the reference line and a travelingdirection line of the vehicle as a direction of the vehicle relative toa lane in which the vehicle is traveling or is scheduled to travel.

While the invention has been described with reference to embodiments,the invention is not limited to the embodiments and can be subjected tovarious modifications and substitutions without departing from the gistof the invention.

What is claimed is:
 1. A vehicle control device comprising: an imageacquirer configured to acquire an image obtained by imaging a spaceoutside of a vehicle; an object detector configured to detect, throughimage processing, a plurality of types of objects including a roadstructure and a moving object shown in the image; a reference linesetter configured to select one type of object from the plurality oftypes of objects and to set a reference line based on a direction of theselected type of object; and a vehicle direction estimator configured toestimate an angle which is formed by the reference line and a travelingdirection line of the vehicle as a direction of the vehicle relative toa lane in which the vehicle is traveling or is scheduled to travel. 2.The vehicle control device according to claim 1, wherein, when theselected type of object is the road structure, the direction of theselected type of object is an extending direction of the road structure,or when the selected type of object is the moving object, the directionof the selected type of object is a traveling direction of the movingobject.
 3. The vehicle control device according to claim 1, wherein thereference line setter selects one type of object from among the detectedplurality of types of objects based on a predetermined priority.
 4. Thevehicle control device according to claim 3, wherein, in thepredetermined priority, a boundary line of the lane in which the vehicleis traveling or is scheduled to travel has the highest priority, aboundary line of an adjacent lane of the lane has the second highestpriority, and another vehicle near the vehicle has the third highestpriority.
 5. The vehicle control device according to claim 1, furthercomprising a driving controller configured to generate a targettrajectory based on the direction of the vehicle estimated by thevehicle direction estimator and to control steering andacceleration/deceleration of the vehicle regardless of operation of adriver of the vehicle such that the vehicle travels along the generatedtarget trajectory.
 6. A vehicle control method which is performed by acomputer mounted on a vehicle, the vehicle control method comprising:acquiring an image obtained by imaging a space outside of a vehicle;detecting, through image processing, a plurality of types of objectsincluding a road structure and a moving object shown in the image;selecting one type of object from the plurality of types of objects andsetting a reference line based on a direction of the selected type ofobject; and estimating an angle which is formed by the reference lineand a traveling direction line of the vehicle as a direction of thevehicle relative to a lane in which the vehicle is traveling or isscheduled to travel.
 7. A computer-readable non-transitory storagemedium storing a program for causing a computer mounted on a vehicle to:acquire an image obtained by imaging a space outside of a vehicle;detect, through image processing, a plurality of types of objectsincluding a road structure and a moving object shown in the image;select one type of object from the plurality of types of objects and seta reference line based on a direction of the selected type of object;and estimate an angle which is formed by the reference line and atraveling direction line of the vehicle as a direction of the vehiclerelative to a lane in which the vehicle is traveling or is scheduled totravel.